Webscraping: (2) Client-side and APIs

Sign-up

We’re going to be downloading economic data from the FRED API. This will require that you first create a user account and then register a personal API key.

External software

Today I’ll be using JSONView, a browser extension that renders JSON output nicely in Chrome and Firefox. (Not needed, but recommended.)

R packages

• New: jsonlite, httr, listviewer, fredr
• Already used: tidyverse, lubridate, hrbrthemes

The httr package already comes bundled with the tidyverse. So you only need to install the following:

## Not run. (Run this manually yourself if you haven't installed these packages yet.)
install.packages(c("jsonlite", "listviewer", "fredr"))

We might as well load the tidyverse now, since we’ll be using it a fair bit today. It’s not necessary, but I’ll also set my preferred ggplot2 theme for the rest of this document.

library(tidyverse)
theme_set(hrbrthemes::theme_ipsum())

Student presentation: APIs

If you’re new to APIs or reading this after the fact, then I recommend this excellent resource from Zapier: An Introduction to APIs. It’s fairly in-depth, but you don’t need to work through the whole thing to get the gist. The summary version is that an API is really just a collection of rules and methods that allow different software applications to interact and share information. This includes not only web servers and browsers, but also software packages like the R libraries we’ve been using.¹ Key concepts include:

• Server: A powerful computer that runs an API.
• Client: A program that exchanges data with a server through an API.
• Protocol: The “etiquette” underlying how computers talk to each other (e.g. HTTP).
• Methods: The “verbs” that clients use to talk with a server. The main one that we’ll be using is GET (i.e. ask a server to retrieve information), but other common methods are POST, PUT and DELETE.
• Requests: What the client asks of the server (see Methods above).
• Response: The server’s response. This includes a Status Code (e.g. “404” if not found, or “200” if successful), a Header (i.e. meta-information about the reponse), and a Body (i.e the actual content that we’re interested in).
• Etc.

A bit more about API endpoints

A key point in all of this is that, in the case of web APIs, we can access information directly from the API database if we can specify the correct URL(s). These URLs are known as an API endpoints.

API endpoints are in many ways similar to the normal website URLs that we’re all used to visiting. For starters, you can navigate to them in your web browser. However, whereas normal websites display information in rich HTML content — pictures, cat videos, nice formatting, etc. — an API endpoint is much less visually appealing. Navigate your browser to an API endpoint and you’ll just see a load of seemingly unformatted text. In truth, what you’re really seeing is (probably) either JSON (JavaScript Object Notation) or XML (Extensible Markup Language).

You don’t need to worry too much about the syntax of JSON and XML. The important thing is that the object in your browser — that load of seemingly unformatted text — is actually very precisely structured and formatted. Moreover, it contains valuable information that we can easily read into R (or Python, Julia, etc.) We just need to know the right API endpoint for the data that we want.

Let’s practice doing this through a few example applications. I’ll start with the simplest case (no API key required, explicit API endpoint) and then work through some more complicated examples.

Do it yourself

As with all APIs, a good place to start is the FRED API developer docs. If you read through these, you’d see that the endpoint path we’re interested in is series/observations. This endpoint “gets the observations or data values for an economic data series”. The endpoint documentation gives a more in-depth discussion, including the various parameters that it accepts.³ However, the parameters that we’ll be focused on here are simply:

• file_type: “json” (Not required, but our preferred type of output.)
• series_id: “GNPCA” (Required. The series data that we want.)
• api_key: “YOUR_API_KEY” (Required. Go and fetch/copy your key now.)

Let’s combine these parameters with the endpoint path to view the data directly in our browser. Head over to https://api.stlouisfed.org/fred/series/observations?series_id=GNPCA&api_key=YOUR_API_KEY&file_type=json, replacing “YOUR_API_KEY” with your actual key. You should see something like the following:

At this point you’re probably tempted to read the JSON object directly into your R environment using the jsonlite::readJSON() function. And this will work. However, that’s not what we’re going to here. Rather, we’re going to go through the httr package. Why? Well, basically because httr comes with a variety of features that allow us to interact more flexibly and securely with web APIs.

Let’s start by defining some convenience variables such as the endpoint path and the parameters (which we’ll store in a list).

endpoint = "series/observations"
params = list(
  api_key= "YOUR_FRED_KEY", ## Change to your own key
  file_type="json", 
  series_id="GNPCA"
  )

Protip: If you want to safely use sensitive information like API keys or passwords — even when sharing content (like these lecture notes) with others — then save it as an R environment variable. You can read more here and here, but something like the below should work. Run this in your console, never in an R Markdown file or other shared document.

Sys.setenv(MY_API_KEY="abcdefghijklmnopqrstuvwxyz0123456789") ## Add your key as an environment variable. (Current session only.)
# usethis::edit_r_environ() ## Open your .Renviron file and add the key. (Persists over sessions.)
readRenviron("~/.Renviron") ## Refresh your .Renviron file

Once that’s done, you can safely assign your key to an object (including within a R Markdown file that you’re going to share) using, say, api_key <- Sys.getenv("MY_API_KEY").

Okay, back to our FRED example. We’ll use the httr::GET() function to request (i.e. download) the data. I’ll assign this to an object called fred.

library(httr)

fred <- 
  httr::GET(
    url = "https://api.stlouisfed.org/", ## Base URL
    path = paste0("fred/", endpoint), ## The API endpoint
    query = params ## Our parameter list
    )

Take a second to print the fred object in your console. What you’ll see is pretty cool; i.e. it’s the actual API response, including the Status Code and Content. Something like:

## Response [https://api.stlouisfed.org/fred/series/observations?api_key=YOUR_API_KEY&file_type=json&series_id=GNPCA]
##   Date: 2019-02-01 00:06
##   Status: 200
##   Content-Type: application/json; charset=UTF-8
##   Size: 9.09 kB

To extract the content (i.e. data) from of this response, I’ll use the httr::content() function. Moreover, we know that this content is a JSON array, so we can convert it to an R object using jsonlite::fromJSON() as we did above. However, we don’t yet know what format it will be in. Okay — SPOILER — it’s going to be a list. I could use the base str() function to delve into the structure of this list. However, I want to introduce you to the listviewer::jsonedit() function, which allows for interactive inspection of list objects.⁴

fred %>% 
  httr::content("text") %>%
  jsonlite::fromJSON() %>%
  listviewer::jsonedit(mode = "view")

Luckily, this particular list object isn’t too complicated. We can see that we’re really interested in the fred$observations sub-element. I’ll re-run most of the above code and then extract this element. I could do this in several ways, but will use the purrr::pluck() function here.

fred <-
  fred %>% 
  httr::content("text") %>%
  jsonlite::fromJSON() %>%
  purrr::pluck("observations") %>% ## Extract the "$observations" list element
  # .$observations %>% ## I could also have used this
  # magrittr::extract("observations") %>% ## Or this
  as_tibble() ## Just for nice formatting
fred

## # A tibble: 0 x 0

Okay! We’ve finally got our data and are nearly ready for some plotting. However, recall that fromJSON() automatically converts everything to characters so I’ll quickly change some variables to dates (using lubridate::ymd()) and numeric.

library(lubridate)

fred <-
  fred %>%
  mutate_at(vars(realtime_start:date), ymd) %>%
  mutate(value = as.numeric(value)) 

Let’s plot this sucker.

fred %>%
  ggplot(aes(date, value)) +
  geom_line() +
  scale_y_continuous(labels = scales::comma) +
  labs(
    x="Date", y="2012 USD (Billions)",
    title="US Real Gross National Product", caption="Source: FRED"
    )

Use a package

One of the great features about the R (and data science community in general) is that someone has probably written a package that does all the heavy API lifting for you. We’ll come across many examples during the remainder of this course, but for the moment I want you to check out the fredr package. How would you access the same GDP data as above using this package? (Hint: See here.)

Locating the hidden API endpoint

Fortunately, there’s a better way: Access the full database of rankings through the API. First we have to find the endpoint, though. Here’s a step-by-step guide of how to that that. It’s fairly tedious, but pretty intuitive once you get the hang of it. You can just skip to the GIF below if you would rather see what I did instead of reading through all the steps.

• Start by inspecting the page. (Ctr+Shift+I in Chrome. Ctrl+Shift+Q in Firefox.)
• Head to the Network tab at the top of the inspect element panel.
• Click on the XHR button.⁷
• Refresh the page (Ctrl+R). This will allow us to see all the web traffic coming to and from the page in our inspect panel.
• Our task now is to scroll these different traffic links and see which one contains the information that we’re after.
• The top traffic link item references a URL called https://cmsapi.pulselive.com/rugby/rankings/mru?language=en&client=pulse. Hmmm. “API” you say? “Rankings” you say? Sounds promising…
• Click on this item and open up the Preview tab.
• In this case, we can see what looks to be the first row of the rankings table (“New Zealand”, etc.)
• To make sure, you can grab that https://cmsapi.pulselive.com/rugby/rankings/mru?language=en&client=pulse, and paste it into our browser (using the JSONView plugin) from before.

Sweet. Looks like we’ve located our API endpoint. As promised, here’s GIF of me walking through these steps in my browser:

Pulling the data into R

Let’s pull the data from the API endpoint into R. Again, I’ll be using jsonlite::readJSON() function.

endpoint <- "https://cmsapi.pulselive.com/rugby/rankings/mru?language=en&client=pulse"
rugby <- fromJSON(endpoint)
str(rugby)

## List of 3
##  $ label    : chr "Mens Rugby Union"
##  $ entries  :'data.frame':   105 obs. of  6 variables:
##   ..$ team       :'data.frame':  105 obs. of  5 variables:
##   .. ..$ id          : int [1:105] 37 36 33 34 39 38 35 46 42 40 ...
##   .. ..$ altId       : logi [1:105] NA NA NA NA NA NA ...
##   .. ..$ name        : chr [1:105] "New Zealand" "Ireland" "Wales" "England" ...
##   .. ..$ abbreviation: chr [1:105] "NZL" "IRE" "WAL" "ENG" ...
##   .. ..$ annotations : logi [1:105] NA NA NA NA NA NA ...
##   ..$ matches    : int [1:105] 209 182 196 189 200 217 177 111 187 182 ...
##   ..$ pts        : num [1:105] 92.5 91.2 87.2 86.2 84.6 ...
##   ..$ pos        : int [1:105] 1 2 3 4 5 6 7 8 9 10 ...
##   ..$ previousPts: num [1:105] 92.5 91.2 87.2 86.2 84.6 ...
##   ..$ previousPos: int [1:105] 1 2 3 4 5 6 7 8 9 10 ...
##  $ effective:List of 3
##   ..$ millis   : num 1.55e+12
##   ..$ gmtOffset: num 0
##   ..$ label    : chr "2019-01-07"

We have a nested list, where what looks to be the main element of interest, $entries, is itself a list.⁸ Let’s extract the $entries element and have a look at its structure. We could use the str() base function, but again the interactivity of listviewer::jsonedit() is hard to beat for complicated list structures.

# str(rugby$entries) ## Base option
listviewer::jsonedit(rugby, mode = "view")

For completeness, let’s take peak at the rugby$entries$team data frame to confirm that it has information that is useful to us.

head(rugby$entries$team)

##   id altId         name abbreviation annotations
## 1 37    NA  New Zealand          NZL          NA
## 2 36    NA      Ireland          IRE          NA
## 3 33    NA        Wales          WAL          NA
## 4 34    NA      England          ENG          NA
## 5 39    NA South Africa          RSA          NA
## 6 38    NA    Australia          AUS          NA

Okay, a clearer picture is starting to emerge. It looks like we can just bind the columns of the rugby$entries$team data frame directly to the other elements of the parent $team “data frame” (actually: “list”). Let’s do that using dplyr::bind_cols() and then clean things up a bit. I’m going to call the resulting data frame rankings.

library(janitor)

rankings <-
  bind_cols(
    rugby$entries$team,
    rugby$entries %>% select(matches:previousPos)
  ) %>%
  clean_names() %>%
  select(-c(id, alt_id, annotations)) %>% ## These columns aren't adding much of interest
  select(pos, pts, everything()) %>% ## Reorder remaining columns
  as_tibble() ## "Enhanced" tidyverse version of a data frame
rankings

## # A tibble: 105 x 7
##      pos   pts name         abbreviation matches previous_pts previous_pos
##    <int> <dbl> <chr>        <chr>          <int>        <dbl>        <int>
##  1     1  92.5 New Zealand  NZL              209         92.5            1
##  2     2  91.2 Ireland      IRE              182         91.2            2
##  3     3  87.2 Wales        WAL              196         87.2            3
##  4     4  86.2 England      ENG              189         86.2            4
##  5     5  84.6 South Africa RSA              200         84.6            5
##  6     6  82.4 Australia    AUS              217         82.4            6
##  7     7  81.8 Scotland     SCO              177         81.8            7
##  8     8  77.9 Fiji         FJI              111         77.9            8
##  9     9  77.3 France       FRA              187         77.3            9
## 10    10  77.0 Argentina    ARG              182         77.0           10
## # … with 95 more rows

BONUS: Get and plot the rankings history

The above looks great, except for the fact that it’s just a single snapshot of the most recent rankings. We are probably more interested in looking back at changes in the ratings over time. For example, back to an era when South Africa wasn’t so kak…

How do we do this? Well, in the spirit of art-vs-science, let’s open up the Inspect window of the rankings page again and start exploring. What happens if we click on the calendar element, say, change the year to “2018” and month to “April”? (Do this yourself.)

This looks promising! Essentially, we get the same API endpoint that we saw previously, but now appended with a date, https://cmsapi.pulselive.com/rugby/rankings/mru?date=2018-05-01&client=pulse. If you were to continue along in this manner — clicking on the website calendar and looking for XHR traffic — you would soon realise that these date suffixes follow a predictable pattern: They are spaced out a week apart and always fall on a Monday. In other words, World Rugby updates its internatinoal rankings table weekly and publishes the results on Mondays.

We now have enough information to write a function that will loop over a set of dates and pull data from the relevant API endpoint. NB: I know we haven’t gotten to the programming section of the course, so don’t worry about the specifics of the next few code chunks. I’ll try to comment my code quite explicitly, but I mostly want you to focus on the big picture.

To start, we need a vector of valid dates to loop over. I’m going to use various functions from the lubridate package to help with this. Note that I’m only to extract a few data points — one observation a year for the last decade or so — since I just want to demonstrate the principle. No need to hammer the host server. (More on that below.)

## We'll look at rankings around Jan 1st each year. I'll use 2004 as an
## arbitrary start year and then proceed until the present year.
start_date <- ymd("2004-01-01")
end_date <- floor_date(today(), unit="years")
dates <- seq(start_date, end_date, by="years")
## Get the nearest Monday to Jan 1st to coincide with rankings release dates.
dates <- floor_date(dates, "week", week_start = getOption("lubridate.week.start", 1))
dates

##  [1] "2003-12-29" "2004-12-27" "2005-12-26" "2007-01-01" "2007-12-31"
##  [6] "2008-12-29" "2009-12-28" "2010-12-27" "2011-12-26" "2012-12-31"
## [11] "2013-12-30" "2014-12-29" "2015-12-28" "2016-12-26" "2018-01-01"
## [16] "2018-12-31"

Next, I’ll write out a function that I’ll call rugby_scrape. This function will take a single argument; namely a date that it will use to construct a new API endpoint during each iteration. Beyond that, it will do pretty much exactly the same things that we did in our previous, manual data scrape. The only other difference is that it will wait three seconds after running (i.e. Sys.sleep(3)). I’m adding this final line to avoid hammering the server with instantaneous requests when we put everything into a loop.

## First remove our existing variables. This is not really necessary, since R is smart enough
## to distinguish named objects in functions from named objects in our global environment.
## But I want to emphasise that we're creating new data here and avoid any confusion.
rm(rugby, rankings, endpoint)

## Now, create the function. I'll call it "rugby_scrape".
rugby_scrape <- 
  function(x) {
    endpoint <- paste0("https://cmsapi.pulselive.com/rugby/rankings/mru?date=", x, "&client=pulse")
    rugby <- fromJSON(endpoint)
    rankings <-
      bind_cols(
        rugby$entries$team,
        rugby$entries %>% select(matches:previousPos)
        ) %>%
      clean_names() %>%
      mutate(date = x) %>% ## New column to keep track of the date 
      select(-c(id, alt_id, annotations)) %>% ## These columns aren't adding much of interest
      select(date, pos, pts, everything()) %>% ## Reorder remaining columns
      as_tibble() ## "Enhanced" tidyverse version of a data frame
    Sys.sleep(3) ## Be nice!
    return(rankings)
  }

Finally, we can now iterate (i.e. loop) over our dates vector, by plugging the values sequentially into our rugby_scrape function. There are a variety of ways to iterate in R, but I’m going to use an lapply() call below.⁹ We’ll then bind everything into a single data frame using dplyr::bind_rows() and name the resulting object rankings_history.

rankings_history <-
  lapply(dates, rugby_scrape) %>% ## Run the iteration
  bind_rows() ## Bind the resulting list of data frames into a single data frame
rankings_history

## # A tibble: 1,569 x 8
##    date         pos   pts name  abbreviation matches previous_pts
##    <date>     <int> <dbl> <chr> <chr>          <int>        <dbl>
##  1 2003-12-29     1  94.0 Engl… ENG               17         92.1
##  2 2003-12-29     2  90.1 New … NZL               17         88.2
##  3 2003-12-29     3  86.6 Aust… AUS               17         88.4
##  4 2003-12-29     4  82.7 Fran… FRA               17         84.7
##  5 2003-12-29     5  81.2 Sout… RSA               15         81.2
##  6 2003-12-29     6  80.5 Irel… IRE               15         80.5
##  7 2003-12-29     7  78.0 Arge… ARG               14         78.0
##  8 2003-12-29     8  76.9 Wales WAL               15         76.9
##  9 2003-12-29     9  76.4 Scot… SCO               15         76.4
## 10 2003-12-29    10  73.5 Samoa SAM               14         73.5
## # … with 1,559 more rows, and 1 more variable: previous_pos <int>

Let’s review what we just did:

• We created a vector of dates — creatively called dates — with observations evenly spaced (about) a year apart, falling on the Monday closest to Jan 1st for that year.
• We then iterated (i.e. looped) over these dates using a function, rugby_scrape, which downloaded and cleaned data from the relevant API endpoint.
• At the end of each iteration, we told R to wait a few seconds before executing the next step. The reason is that R can execute these steps much, much quicker than we could ever type them manually. It probably doesn’t matter for this example, but you can easily “overwhelm” a host server by hammering it with a loop of automated requests. (Or, just as likely: They have safeguards against this type of behaviour and will start denying your requests as a suspected malicious attack.) The “be nice” motto is important to remember when scraping API data.
• Note that each run of our iteration will have produced a separate data frame, which lapply() by default appends into a list. We used dplyr::bind_rows() to bid these separate data frames into a single data frame.

Okay! Let’s plot the data and highlight a select few countries in the process.

teams <- c("NZL", "RSA", "ENG", "JPN")
team_cols <- c("NZL"="black", "RSA"="#4DAF4A", "ENG"="#377EB8", "JPN" = "red")

rankings_history %>%
  ggplot(aes(x=date, y=pts, group=abbreviation)) +
  geom_line(col = "grey") + 
  geom_line(
    data = rankings_history %>% filter(abbreviation %in% teams), 
    aes(col=fct_reorder2(abbreviation, date, pts)),
    lwd = 1
    ) +
  scale_color_manual(values = team_cols) +
  labs(
    x = "Date", y = "Points", 
    title = "International rugby rankings", caption = "Source: World Rugby"
    ) +
  theme(legend.title = element_blank())

It was a good time to be a South African rugby supporter around 2007–2009. These days? Not so much. OTOH, New Zealand’s extended dominance in the global game is extraordinary. (Especially given its tiny population size.) They truly do have a legimate claim to being the greatest international team in the history of professional sport. Which is a good reason to link to the best ever haka.